Quantitative second-harmonic generation imaging is employed to assess stromal collagen in normal, hyperplastic, dysplastic, and malignant breast tissues. The cellular scale organization is quantified using Fourier transform-second harmonic generation imaging (FT-SHG), while the molecular scale organization is quantified using polarization-resolved second-harmonic generation measurements (P-SHG). In the case of FT-SHG, we apply a parameter that quantifies the regularity in collagen fiber orientation and find that malignant tissue contains locally aligned fibers compared to other tissue conditions. Alternatively, using P-SHG we calculate the ratio of tensor elements (d15/d31, d22/d31, and d33/d31) of the second-order susceptibility χ2 for collagen fibers in breast biopsies. In particular, d15/d31 shows potential differences across the tissue pathology. We also find that trigonal symmetry (3m) is a more appropriate model to describe collagen fibers in malignant tissues as opposed to the conventionally used hexagonal symmetry (C6). This novel method of targeting collagen fibers using a combination of two quantitative SHG techniques, FT-SHG and P-SHG, holds promise for breast tissue analysis and applications to characterizing cancer in a manner that is compatible with clinical practice.